Fix template path validator for composable integration packages by teresaromero · Pull Request #1108 · elastic/package-spec

teresaromero · 2026-03-11T10:22:39Z

What does this PR do?

Fixes the ValidateIntegrationPolicyTemplates semantic validator to correctly handle integration packages that use the 3.6.0 composability feature (package references in policy template inputs and data stream streams).

Adds three example test packages that cover all possible composable cases defined in the spec.

Why is it important?

While working on enabling composable integrations in elastic-package, validation was failing on any integration package that used package: references in policy template inputs or data stream streams — even when the package was otherwise valid.

The root cause is a zero-value collision in the Go structs. When a policy template input uses package instead of type, the Type field unmarshals to "". Similarly, when a data stream stream uses package instead of input, the Input field is also "". The validator matched these two empty strings, then tried to open data_stream/<name>/agent/stream — a directory that doesn't exist for composable streams (they inherit all configuration from the referenced input package), producing a false no such file or directory error.

The fix adds a Package field to both structs and skips template path validation whenever a package reference is present.

Checklist

I have added test packages to test/packages that prove my change is effective.
I have added an entry in spec/changelog.yml.

Related issues

Relates to the package dependencies feature: Add definitions for package dependencies #1071

Made with Cursor

…dd example test packages When a policy template input uses `package` instead of `type`, or a data stream stream uses `package` instead of `input`, the Go zero value for the missing string field is `""`. The existing template path validator matched these empty strings against each other, then tried to open a non-existent `agent/stream` directory, causing a false validation failure on any composable integration package. Fix by skipping template path validation for inputs and streams that carry a `package` reference, since configuration is inherited from the referenced input package and no local agent stream template exists. Add three example test packages covering all composable cases from the spec: - good_composable_pt_inputs: all policy template inputs use package reference - good_composable_ds_streams: all data stream streams use package reference - good_composable_multi_package: multiple input packages in both locations, plus requires.content dependencies Made-with: Cursor

coderabbitai · 2026-03-11T10:26:38Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 83649c28-ee6c-4c91-adc6-0b10ca63a7d7

📥 Commits

Reviewing files that changed from the base of the PR and between 294f278 and 11b7e83.

📒 Files selected for processing (21)

code/go/internal/validator/semantic/validate_integration_policy_template_path.go
code/go/pkg/validator/validator_test.go
spec/changelog.yml
test/packages/good_composable_ds_streams/changelog.yml
test/packages/good_composable_ds_streams/data_stream/logs/fields/fields.yml
test/packages/good_composable_ds_streams/data_stream/logs/manifest.yml
test/packages/good_composable_ds_streams/docs/README.md
test/packages/good_composable_ds_streams/manifest.yml
test/packages/good_composable_multi_package/changelog.yml
test/packages/good_composable_multi_package/data_stream/db_metrics/fields/fields.yml
test/packages/good_composable_multi_package/data_stream/db_metrics/manifest.yml
test/packages/good_composable_multi_package/data_stream/web_logs/fields/fields.yml
test/packages/good_composable_multi_package/data_stream/web_logs/manifest.yml
test/packages/good_composable_multi_package/docs/README.md
test/packages/good_composable_multi_package/manifest.yml
test/packages/good_composable_pt_inputs/changelog.yml
test/packages/good_composable_pt_inputs/data_stream/metrics/agent/stream/stream.yml.hbs
test/packages/good_composable_pt_inputs/data_stream/metrics/fields/fields.yml
test/packages/good_composable_pt_inputs/data_stream/metrics/manifest.yml
test/packages/good_composable_pt_inputs/docs/README.md
test/packages/good_composable_pt_inputs/manifest.yml

📝 Walkthrough

Walkthrough

This PR introduces composable package support to the validator. A new Package field is added to policy template inputs and data streams. The validation logic in the semantic validator is updated to skip local template_path validation checks for inputs and streams that reference a package. Three new test packages demonstrate composable scenarios: one with policy template inputs referencing packages, one with data stream streams referencing packages, and one using multiple input packages across different data streams.

Possibly related PRs

Adjust property names for package dependencies to original proposal #1102: Modifies the semantic validator to recognize and handle the "package" property for package-based references, directly supporting the composable package feature introduced in this PR.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment

🛠️ Update Documentation: Commit on current branch
🛠️ Update Documentation: Create PR

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

elasticmachine · 2026-03-11T10:38:15Z

💚 Build Succeeded

Buildkite Build
Commit: 11b7e83

jsoriano

This looks like another case for #549, where we would need different validations for source and built packages.

jsoriano · 2026-03-11T11:24:01Z

code/go/pkg/validator/validator_test.go

+		"good_composable_pt_inputs":              {},
+		"good_composable_ds_streams":             {},


Nit.

Suggested change

"good_composable_pt_inputs": {},

"good_composable_ds_streams": {},

"good_composable_policy_template_inputs": {},

"good_composable_data_stream_streams": {},

jsoriano · 2026-03-11T11:24:41Z

spec/changelog.yml

      link: https://github.com/elastic/package-spec/pull/1092
+    - description: Fix template path validator to skip policy template inputs and data stream streams that use package references.
+      type: bugfix
+      link: https://github.com/elastic/package-spec/pull/1108


No need to add changelog for this as package references are also introduced in 3.6.

jsoriano · 2026-03-11T11:25:39Z

code/go/pkg/validator/validator_test.go

+		"good_composable_pt_inputs":              {},
+		"good_composable_ds_streams":             {},
+		"good_composable_multi_package":          {},


Could these cases be added to existing good_packages instead of adding a test package for each one?

Btw, good_composable_pt_inputs and good_composable_ds_streams also pass with the code in main. Is this expected?

jsoriano · 2026-03-11T11:28:56Z

test/packages/good_composable_ds_streams/data_stream/logs/manifest.yml

+title: Web server logs
+type: logs
+streams:
+  - package: filelog_otel


Is this package correct in a real case? This input is not used on any policy template 🤔 Only logfile is used.

i dont know if is a real-case, but is a possible case...
when the policy template has certain imputs and the required package is at a data stream, there is no "type: logfile" linking policy template with streams. is this possible?

jsoriano · 2026-03-11T11:40:20Z

code/go/internal/validator/semantic/validate_integration_policy_template_path.go

 func validateIntegrationPackagePolicyTemplate(fsys fspath.FS, policyTemplate integrationPolicyTemplate, dsManifestMap map[string]dataStreamManifest) error {
 	for _, input := range policyTemplate.Inputs {
+		// inputs using package references inherit configuration from the input package,
+		// so there is no local agent stream template to validate.


This will be still validated on the built package, right?

i dont understand the question :S

Before building, this validation takes place. Here we are looking for the template_path so we check if exist. Data streams and policy templates are "linked" by the input.type -> stream.input;

when using package, the validation does not find template path, and injects empty "type" into the stream validation, to eventually find the template path.. when in the stream template validation, it finds that package is used, so input is also ""; empty == empty; it looks for the default stream.yml.hbs and it does not find it.

Shall this "connection" be fixed? perhaps i took a wrong assumption

I think the code is fine, I was only asking for confirmation.

elastic-package build validates the built package after building it. In this case built packages will have streams here even if their source code didn't have them.

Umm, actually I see that streams merge is not described on any issue, apart from variables and template paths. I will create one.

elastic/elastic-package#3380

jsoriano · 2026-03-11T11:45:26Z

test/packages/good_composable_pt_inputs/data_stream/metrics/manifest.yml

+title: Database metrics
+type: metrics
+streams:
+  - input: logfile


This input is not used on any policy template. Would this work as a real package?

same as the other example, out of the spec this is possible for a package, i created them based on what possibilities a package can have. is there a requirement for data streams and inputs from policy templates to be "conected". i remember the conexion is between the input type (policy) and the input (Stream). they should appear allways? i have a gap here.

if a policy template has a package, with a template, does it make sense to have a data stream with templates too?

teresaromero added 2 commits March 11, 2026 11:21

Update changelog to correct link

11b7e83

teresaromero marked this pull request as ready for review March 11, 2026 10:23

teresaromero requested a review from a team as a code owner March 11, 2026 10:23

jsoriano reviewed Mar 11, 2026

View reviewed changes

teresaromero closed this Mar 16, 2026

		"good_composable_pt_inputs": {},
		"good_composable_ds_streams": {},

Conversation

teresaromero commented Mar 11, 2026

What does this PR do?

Why is it important?

Checklist

Related issues

Uh oh!

coderabbitai bot commented Mar 11, 2026

Walkthrough

Possibly related PRs

Uh oh!

elasticmachine commented Mar 11, 2026

💚 Build Succeeded

Uh oh!

jsoriano left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

teresaromero Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

teresaromero Mar 13, 2026 •

edited

Loading